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ABSTRACT 



The purpose of this study was to determine the measurement 
comparability of paper- and-pencil and multimedia versions of two career 
assessment components in the DISCOVER multimedia career and education 
planning system. Students in grade 9 (N=606) and 11 (N=416) completed 
paper-and-pencil and Compact Disc- interactive (CD-i) versions of the Unisex 
Edition of the American College Testing Inventory (UNIACT) and the Inventory 
of Work-Relevant Abilities (IWRA) . Measurement comparability of the versions 
was evaluated by comparing mean scale score differences, correlations between 
corresponding scales, and patterns of scale intercorrelations. In addition, 
targeted factor analyses were run for UNICAT and principal components 
analysis for UNIACT and IWRA. The results indicate that the use of pictures 
and voice-overs in the multimedia versions of the assessments yields scores 
comparable to those obtained with the paper-and-pencil versions. Appendixes 
contain the career guidance survey and a discussion of test equating. 
(Contains 4 figures, 18 tables, and 52 references.) (Author/SLD) 
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Abstract 



The purpose of this study was to determine the measurement comparability of paper-and- 
pencil and multimedia versions of two career assessment components in DISCOVER. Grade 9 
(N = 606) and 11 (N = 416) students completed paper-and-pencil and Compact Disc-interactive 
(CD-i) versions of the Unisex Edition of the ACT Interest Inventory (UNIACT) and the 
Inventory of Work-Relevant Abilities (IWRA). Measurement comparability of the versions was 
evaluated by comparing mean scale score differences, correlations between corresponding scales, 
and patterns of scale intercorrelations. In addition, targeted factor analyses were run for 
UNIACT and principal component analyses for UNIACT and IWRA. The results indicate that 
the use of pictures and voice-overs in the multimedia versions of the assessments yield scores 
comparable to those obtained with the paper-and-pencil versions. 
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Measurement Comparability of Paper-and-Pencil 
and Multimedia Vocational Assessments 

There have been widespread changes in electronic and computer hardware, including the 
advent of laser disc technology. With the innovation and flexibility of Compact Disc-interactive 
(CD-i) and CD-ROM technologies, a range of multisensory options is possible; printed text can be 
combined with color photographs, CD-quality sound, and full-motion video sequences. These new 
technologies, when incorporated in computer-based career and educational planning systems, have 
the potential of offering counselees an enhanced career exploration experience (Sampson, 1997). 

DISCOVER Multimedia (ACT, 1995a) is a career and educational planning system that 
operates on a CD-i player. DISCOVER for Windows (ACT, 1997) is a computer-based (CD- 
ROM) vocational guidance and information software system. The systems' two assessment 
components, the Revised Unisex Edition of the ACT Interest Inventory (UNIACT) and the 
Inventory of Work-Relevant Abilities (IWRA), add pictures and voice-overs to item text based on 
the established paper-and-pencil versions of these assessments. Determining whether the paper- 
and-pencil-based norms and validity data are applicable to the multimedia versions is worthy of 
empirical attention. 

Measurement Equivalence of Computerized Versions of Paper-and-Pencil Tests 

In 1986, the American Psychological Association (APA) published Guidelines for 
Computer-Based Tests and Interpretations to deal with such issues as the equivalence of 
computerized and paper-and-pencil tests. Equivalence is defined in Guideline 16 as: 

Scores from conventional and computer administrations may be 
considered equivalent when (a) the rank orders of the scores of 
individuals tested in alternative modes closely approximate each 
other, and (b) the means, dispersions, and shapes of the score 
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distributions are approximately the same, or have been made 
approximately the same by rescaling the scores from the computer 
mode. (APA, 1986, p. 18) 

According to Green (1991), cross-mode comparability studies reported in the literature 
typically involve comparing cross-mode scale correlations to determine whether the tests are 
measuring the same constructs. Low cross-mode correlations suggest that the tests are measuring 
different constructs. If there are high cross-mode scale correlations, comparisons of the means, 
standard deviations, and shapes of the score distributions are warranted. If the means, standard 
deviations, and shapes of distributions are different, the computer scores can be transformed or 
rescaled to the paper-and-pencil scores. Establishing test equivalence (and rescaling, if necessary,) 
allows computer scores to be interpreted using norms from the paper-and-pencil test. Furthermore, 
evidence of the validity of the paper-and-pencil version can be generalized to the computerized 
version. 

In the area of noncognitive testing, equivalency studies involving personality inventories 
have yielded mixed results. Mazzeo and Harvey (1988) reviewed studies comparing the 
psychometric properties of computerized and paper-and-pencil tests. In the section of their review 
dealing with untimed personality inventories with single-screen items, they reviewed five studies 
that reported lower mean scores for the computerized versions of the Minnesota Multiphasic 
Personality Inventory and the California Personality Inventory. Mazzeo and Harvey noted that, 
although respondents could omit items in the paper-and-pencil mode, they were forced to answer all 
items in the computerized mode. They concluded that the observed mean score differences were 
due, in part, to the differential rate at which items are omitted. Equivalence was found in three 
studies involving five different psychological measures (e.g., State-Trait Anxiety Inventory, Beck 
Depression Inventory). 




10 



3 



The results of equivalency studies involving career assessments have been unequivocal. In 
1984, Brown (as cited in Hansen, 1987) examined profiles based on paper-and-pencil and 
computerized versions of the Strong-Campbell Interest Inventory (SCH) and found no significant 
mode of administration differences. O'Shea (1987) and Kapes and Vansickle (1992) compared 
computer and paper-and-pencil versions of the Harrington-0 'Shea Career Decision-Making System 
(CDM). O'Shea found no significant mode of administration differences in means, standard 
deviations, or three-point codes. Kapes and Vansickle also found no systematic differences in 
means, standard deviations, or two-point codes; however, they did find that the computerized 
version was more reliable. Reardon and Longhead (1988) conducted an equivalency study of 
computer and paper-and-pencil versions of the Self-Directed Search. The cross-mode correlations 
for the six scales ranged from a low of .86 to a high of .94. Across modes, there were no significant 
differences in mean scores found for the six scale scores. Smularly, Vansickle, Kimmel, and Kapes 
(1989) and Vansickle and Kapes (1993) examined the equivalency of paper-and-pencil and 
computerized versions of the SCH. Results of both studies indicated that the two modes were 
equivalent; however, as discovered by Kapes and Vansickle with the CDM, the computerized 
version of the SCH was more reliable. 

There are two noteworthy aspects of these equivalency studies. First, they compared paper- 
and-pencil and computerized versions of tests that did not include pictures or voice-overs. Second, 
most of these authors examined numerical score comparability by focusing on the scale mean and 
standard deviation differences. 

In more recent equivalency studies (Van de Vijver & Harsveld, 1994; King & Miles, 1995) 
and reviews (Bugbee, 1996), the concept of equivalence has been further defined; empirically 
establishing that paper-and-pencil and computer versions of a test are measuring the same construct 
has emerged as a primary concern. According to Van de Vijver and Harsveld, computerized and 
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paper-and-pencil versions measure the same psychological construct(s) if they are structurally 
equivalent. They suggest that equivalency studies need to first assess the degree of factorial 
similarity by linear structural models or by a factor analysis of the item correlation matrix followed 
by a targeted rotation. In the latter approach, structural or qualitative equivalency is determined by 
comparing the number of factors and the factor loadings across administration modes. Given 
factorial similarity for the two modes of administration, evidence of numerical score comparability 
or quantitative equivalence can then be considered. 

King and Miles' (1995) define measurement equivalence in terms of two elements: constant 
conceptual domain and constant calibration. Constant conceptual domain is assessed by 
establishing the two instrument versions have the same number of underlying factors. Constant 
calibration is assessed by testing for equality of factor loadings across administration modes. 
According to King and Miles, the former provides evidence the two versions measure the same 
construct while the latter provides evidence the two versions measure the same construct to the 
same degree. Furthermore, they suggest that comparison of mean score differences, without 
establishing evidence of a constant conceptual domain and constant calibration, is not meaningful 
and can be misleading. 

Use of Pictures and Audio in Interest Inventories 

DISCOVER Multimedia appears to be the first career planning system to administer interest 
and ability self-estimate items with accompanying color photographs and voice-overs. 
Nevertheless, the use of pictures and audio in interest assessment is not new. 

Ammons, Butler, and Herzig (1950) developed a pioneering set of pictures portraying a 
variety of human activities in their study of vocational preferences. The approach was informal and 
involved the subject's projective statements about the pictures. The Geist Picture Interest Inventory 
(Geist, 1959) was the first formal pictorial interest inventory. It was developed in 1952 as an 
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alternative to verbal inventories that require a high level of verbal proficiency. Since then, a 
number of pictorial inventories have been developed (see Table 1). These instruments use line 
drawings, photographs, slides, or videos of tools, work environments, or individuals engaged in 
various work activities. Either items are presented in a forced-choice format or the counselee 
expresses a preference for each activity on a Likert scale. As noted in Table 1, many of the pictorial 
interest inventories were developed for special populations experiencing low reading ability^ others 
were developed for the general population but are also described as appropriate for non-readers. 
The underlying premise is that pictorial representation of work environments, tools, or workers 
performing tasks is closer to real life (Geist, 1959), is less ambiguous (Jastak & Jastak, 1979), and 
is less open to interpretation (Becker, 1981). 

For most of the assessments in Table 1, pictures are presented with no accompanying text. 
Two inventories combine written text or audible statements with pictures or videos. The Interest- 
Based Career Decision (IBCD; Peterson, 1994) inventory presents both pictures and written text; 
the pictures are included to augment the written items (Jim Roberts, personal communication 
6/24/96). The 40-minute live action video in the Career Assessment Battery (CAB; Piney 
Mountain Press, 1991) was developed to correspond to the assessment used in the Kentucky State 
Career Information Systems. Information about the development of the video or its correspondence 
to the original assessment is not provided in either the CAB technical manual or the CAB Technical 
Brie/ (Piney Mountain Press, 1995). 

Tetreau and Trahan (1983), developers of the Tetreau-Trahan Visual Interest Test (TTVIT: 
Tetreau & Trahan, 1986), conducted a study to determine the advantages of a pictorial versus verbal 
(i.e., written, paper-and-pencil) method of assessing interests. They developed verbal and pictorial 
forms that were parallel in content at the item level. The pictorial form consisted of 102 color 
slides of various occupations. The two forms were administered to 451 high school and college 
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students. Tetreau and Trahan reported that the alpha coefficients and mean scores obtained were 
"rather systematically, if slightly, higher for the TTVIT than for its verbal version, the differences 
being noticeably higher in the case of the female students" (p. 7). Corresponding scale correlations 
for the two modes yielded coefficients ranging from .78 to .91. Tetreau and Trahan concluded that 
the two forms measure "essentially the same dimensions" (p. 7). Information about the factor 
structure of the pictorial version is provided in the test manual; however, there is no information 
about the factor structure of the verbal version. 

Although pictures have been used extensively in vocational assessments, very few 
assessments combine pictures with written text. Developers of those assessments have not 
conducted and reported equivalence studies. The Tetreau and Trahan (1983) study has peripheral 
relevance to the current study; they provided evidence of the quantitative equivalence of a pictorial 
interest inventory and a corresponding verbal version. The larger issue of whether introducing 
pictures and voice-overs to written text affects the psychometric properties (e.g., validity and 
reliability) of an assessment has not been addressed empirically. 

Purposes of the Study 

The primary purpose of this study was to evaluate the measurement comparability of the 
paper-and-pencil and multimedia versions of UNIACT and IWRA. Measurement comparability 
was evaluated by comparing correlations between corresponding scales, patterns of scale 
intercorrelations, and mean scale scores. In addition, targeted factor analyses were run for UNIACT 
and principal component analyses for UNIACT and IWRA. Given evidence of qualitative and 
quantitative comparability for the two modes of presentation, the validity evidence available for the 
paper-and-pencil versions of these instruments would be applicable to the multimedia versions. 

The secondary purpose of the study pertained to norming of the multimedia versions of the 
two inventories. The scoring of these assessments involves the use of normative or scaling data 
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gathered via the administration of paper-and-pencil versions of the assessments to nationally 
representative samples (Swaney, 1995). Given evidence of measurement comparability, the data 
collected in this study can be used to equate multimedia scores to the existing paper-and-pencil- 
based scores, permitting paper-and-pencil-based norms to be used in scoring the multimedia 
versions of the two inventories. 

Method 

Pilot Study 

In March 1993, a pilot study was conducted using 9th-grade students enrolled in a Spanish 
class in a rural eastern Iowa school. The purposes of the pilot study were to (a) evaluate the 
administration instructions, (b) determine the time needed for students to respond to name, grade, 
and other background information items, and (c) determine the time needed for students to respond 
to CD-i items. The CD-i versions of UNIACT and IWRA were administered. In-class 
observations and tape recordings of the administration were used to estimate the per-item and total 
time needed to administer the CD-i versions of the two inventories. 

Samples 

Data collection was conducted in two phases. In the spring of 1993, data were collected 
from 9th-grade students. Nine of 12 Iowa schools contacted agreed to participate in the study. The 
9th-grade sample of schools (see Table 2) included one urban alternative school, one urban 
parochial school, four urban public schools, and three rural public schools. 

In the fall of 1993, data were collected from 1 Ith-grade students. Four of the eight Iowa 
schools contacted agreed to participate in the study. The 1 Ith-grade sample of schools (see Table 
2) included one urban alternative school, one urban parochial school, and two suburban public 
schools. One of the suburban schools, due to its location, also enrolls rural and urban students. 
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In both phases of the study, school officials selected the classes (e.g., 9th-grade English, 
1 Ith-grade history) that would participate. As shown in Table 2, intact classes were assigned to one 
of two treatment groups (X and Y) described subsequently. In general, assignment of classes to 
groups alternated sequentially by class period. When school officials provided classes that varied on 
the basis of academic ability, there was an attempt to balance out the assignment to the two groups 
(X and Y) within and across schools. Students enrolled in those classes and present for both testing 
dates— regardless of grade level — were included in the 9th-grade (N = 606) and 1 Ith-grade (N = 
416) samples. The 9th-grade sample included 603 Grade 9 students and one each 8th-, 10th-, and 
1 Ith-grade students. The 1 Ith-grade sample included 376 Grade 11 students and 40 Grade 12 
students. Results of cross-tab analyses revealed the absence of a self-selection bias in the 
assignment of the 40 Grade 12 students to treatment groups. Table 3 summarizes selected 
characteristics of the samples. 

Instruments 

Paper-and-pencil versions. UNIACT (Swaney, 1995; see items in Appendix A) is a 90- 
item interest inventory designed to help persons identify personally relevant career options. Items 
emphasize work-relevant activities that are familiar to persons, either through participation or 
observation. A three-choice response format (like, indifferent, dislike) is used. The six 15-item 
scales correspond to Holland's (1997) six types. UNIACT scale titles (and corresponding Holland 
types) are Science (Investigative), Arts (Artistic), Social Service (Social), Business Contact 
(Enterprising), Business Operations (Conventional), and Technical (Realistic). Scores on the six 
scales are reported as stanines based on national norms. Internal consistency (Cronbach alpha) 
reliability estimates across UNIACT's six scales for a nationally representative sample of Grade 10 
students ranged from .86 to .92 (median = .88) for males, and from .84 to .92 (median = .87) for 
females (Swaney, 1995). Grade 8 and 12 nationally representative samples yielded nearly identical 
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results to those for Grsde 10. UNXA.CT vtiliciity dsts include ffoll^nd 3“letter codes for more th^n 
79,000 persons in 648 career groups. Swaney (1995) provides specifics regarding UNIACT scale 
reliability and stability; convergent, discriminant, and criterion-related validity; and scoring. 

IWRA (see Appendix A) obtains informed self-estimates for 15 abilities identified by 
research as important to occupational differentiation and success (ACT, 1999). The Career 
Planning Survey Technical Manual (ACT, 1999) describes how these abilities were identified and 
assigned to Holland’s six types. Examples include numerical, mechanical, sales, leadership, and 
organizational abilities. Each ability is accompanied by a brief definition and examples of related 
activities. Respondents rate each ability on a scale of 1 to 5, with 1 = low (bottom 10%), 2 = below 
average (lower 25%), 3 = average (middle 50%), 4 = above average (upper 25%), and 5 = high 
(top 10%). The six IWRA scales have titles identical to those listed for UNIACT (see above). Four 
abilities are assigned to each scale, and 5 of the 15 abilities are assigned to more than one scale. 
Theory and research supporting these assignments is provided by Prediger (1992). Stanine-like 
scores (expressed to two digits) are provided for each scale through a validity-oriented scaling 
procedure described by Swaney (1987). Validity evidence supporting the use of IWRA in career 
counseling is described in the Career Planning Survey Technical Manual (ACT, 1999). 

Multimedia versions. The CD-i versions of UNIACT and IWRA were presented on a 
television screen via a CD-i player. Because televisions used in the study were supplied by the 
schools, screen size varied from school to school but was never smaller than 21 inches. 

For each UNIACT item, one visual representation of the activity is presented along with the 
item text on the screen. A voice-over reads the item text. For example, a visual representation of a 
young man engaged in calligraphy appears to the right of the text Sketch and draw pictures. 

For each IWRA item, four visual representations for each ability appear sequentially on the 
screen at 3-second intervals as a voice-over reads the text. The visual representation appears to the 
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left of the text. Response options are displayed across the bottom of the screen. As an example, for 
the ability Creative/Artistic, the text reads Drawing, painting, playing a musical instrument, acting, 
dancing, etc. The following visual representations of the ability flash, one at a time, on the screen: 
A dancer dressed in a leotard dancing; painter painting a picture; sculptor working on a piece of 
sculpture; potter sitting at the wheel shaping a piece of pottery. The fourth and final visual 
representation for each item remains on the screen until the next item is selected. 

Research Design 

Figure 1 shows the designs used in collecting the 9th- and llth-grade data. All students 
participating in the study completed the paper-and-pencil and CD-i versions of UNIACT and 
IWRA. Mode of presentation (CD-i and paper-and-pencil) for the two inventories was 
counterbalanced across Groups X and Y. Instrument sequence differed for the 9th and 11th 
graders. For Grade 9 students, instrument sequence was initially held constant (i.e., students 
completed UNIACT before IWRA). The research designed was modified (see Figure 1) after the 
ACT representative noted a substantial difference in individual reading speed. For approximately 
half of the sessions, the CD-I inventory was presented before the paper-and-pencil inventory. This 
modification affected only the instrument sequence, not the particular inventories planned for that 
session. Analyses of the Grade 9 data collected according to the initial research design did not 
show an instrument sequence effect. As shown in Figure 1, the modified Grade 9 research design 
was used in collecting the Grade 1 1 data. 

Procedures and Administration 

A school counselor or the classroom teacher introduced the ACT representative at the 
beginning of the first testing session in each classroom. The ACT representative told the students 
that ACT had two new multimedia vocational assessments that they were introducing to the 9th- or 
1 Ith-grade students in their school. For their participation, students were told they would receive 
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score reports for use in career planning. The representative provided each student with a copy of 
the Career Guidance Survey (CGS; see Appendix A), a machine-scorable answer sheet, and a 
pencil. Following directions on the front page of the CGS, students filled in their name, address, 
and other background questions on the answer sheet. Students were cautioned not to turn to other 
sections of the CGS until told to do so. Next, the students were shown the appropriate section of 
the answer sheet for recording their responses to one of the two inventories. (See Figure 1 for order 
and modes of presentation for the two inventories.) For CD-i presentations, the ACT representative 
asked the students to please look at the picture(s) for each item prior to marking their response on 
the answer sheet. The representative observed to make sure they finished recording their response 
before the next item was presented. When necessary, students were reminded to look at the screen 
before responding. After all of the students had recorded their response to the last item, those 
students who completed the CD-I version first were asked to turn to the appropriate page in the 
CGS and complete the other inventory. After all the answer sheets and CGSs had been collected, 
the representative thanked the students for their cooperation and indicated that they would be back 
for some additional testing in a week or so. 

Approximately one to two weeks later, an ACT representative administered the two 
inventories in the appropriate order and mode as shown in Figure 1. After all of the answer sheets 
and CGSs had been collected, the ACT representative explained the purpose of the study and 
answered any questions. Subsequently, score reports based on the paper-and-pencil versions of 
UNIACT and IWRA were sent with interpretive materials to school counselors for use with the 
students. 
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Results and Discussion 

Preliminary Analyses 

Group and time-of-presentation (see research design in Figure 1) differences were examined 
to determine if the data for Groups X and Y could be collapsed, thus providing the maximum 
number of cases to determine the psychometric characteristics of the CD-i versions of the two 
inventories as well as the extent to which scores from the CD-i versions are related to scores from 
the paper-and-pencil versions. Tables 4 and 5 show the group and time-of-presentation mean scale 
score differences for UNIACT and IWRA at both grade levels. 

The Grade 9 UNIACT group and time-of-presentation mean scale score differences are 
shown in Table 4, columns 5 and 10. For the six scales, the mean differences ranged from -0.6 to 
0.3 for the CD-i presentation and -0.5 to 0.4 for the paper-and-pencil presentation. The mean 
absolute differences across the six scales were 0.3 for both modes of presentation. IWRA job 
cluster score mean differences (Table 5) ranged from -1.7 to 2.6 for the CD-i presentation and -3.7 
to -0.4 for the paper-and-pencil presentation. The corresponding mean absolute differences for the 
six scales were 1.7 and 2.4, respectively. Comparing these differences to the corresponding time 
and group scale standard deviations (columns 2, 4, 7, and 9) provides a perspective of the 
magnitude of the differences. For example, the largest Grade 9 group and time-of-presentation 
mean absolute difference was 3.7; this difference was approximately one-fourth of the 
corresponding Business Contact (BC) group and time-of-presentation (paper-and-pencil) standard 
deviations of 14.7 and 15.6. 

The Grade 1 1 group and time-of-presentation mean scale scores are also shown in columns 
5 and 10 of Tables 4 and 5. For the six UNIACT scales, the mean differences were extremely 
small. The mean absolute difference across the six scales was 0.2 for both modes of presentation. 
IWRA job cluster score mean differences for group and time-of-presentation ranged from -1.8 to 
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3.6 for the CD-i presentation and from -3.3 to 0.6 for the paper-and-pencil presentation. The mean 
absolute difference for the six scales was 2.3 (CD-i presentation) and 2.2 (paper-and-pencil 
presentation). Again, when compared to the standard deviations for group and time-of-presentation 
(columns 2, 4, 7, and 9), these differences were small. 

Given the small group and time-of-presentation mean differences, the data were collapsed 
across group and time-of-presentation for all subsequent analyses. 

Qualitative Comparability 

Scale intercorrelations. Cross-mode comparabiUty was assessed by the correlations 
between corresponding scales. Scale intercorrelations simultaneously index the similarity of the 
rankings of examinees across modes and the degree of linearity of the relation between the two sets 
of scores. A low correlation would indicate nonlinearity or a cross mode difference in the 
constructs being assessed. As shown in Table 6, UNIACT cross-mode correlations ranged from .79 
to .87 for Grade 9 and from .78 to .90 for Grade 11. For IWRA, the cross-mode correlation 
coefficients ranged from .71 to .78 and from .69 to .78 for Grades 9 and 1 1, respectively. The lower 
coefficients for the IWRA scales may have occurred because there are only four items per scale. 
The magnitudes of these correlation coefficients provide evidence that the two modes of 
presentation are assessing the same constructs. 

UNIACT and IWRA scales are based on the typology and hexagonal structure of interests 
described in Holland's (1997) theory of careers. We would expect interest scales adjacent to each 
other on Holland's hexagon to be more highly correlated than non-adjacent scales, and scales 
located on opposite sides of the hexagon to have the lowest correlations. Intercorrelations among 
the six UNIACT scales for both paper-and-pencil and CD-i modes are shown in Table 7. The 
pattern of relationships for both modes was highly similar. The relative magnitudes of the adjacent, 
non-adjacent, and diagonal intercorrelations were generally in accord with Holland's theory. 
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Intercorrelations among IWRA job cluster scales are shown in Table 8. Ability 
intercorrelations were higher than interest intercorrelations, probably because 5 of the 15 abilities 
are assigned to more than one scale. The pattern of relationships for both modes was similar. 
Noteworthy departures from the hexagonal model for scale intercorrelations occurred for the 
Science (SC) scale. For both modes and grades, correlations between the SC scale and non- 
adjacent Business Contact (BC) and Business Operations (BO) scales were as high, or higher, than 
correlations between the Science (SC) scale and adjacent Arts (AR) and Technical (TE) scales. 
Reasons for this finding are unclear. 

Factor structure. The stmctural or qualitative comparability of the CD-i and paper-and- 
pencil versions of UNIACT was examined by comparison of the theory-based dimensions 
underlying UNIACT. Research indicates that two theory-based dimensions, depicted in Figure 2, 
underlie Holland's two-dimensional hexagon. Empirical support for data/ideas and things/people 
work task dimensions is summarized by Prediger (1996) and Rounds (1995). 

Cooley and Lohnes' (1971) targeted factor extraction procedure was used to compare the 
dimensions (principal component) from a set of intercorrelations. No factor rotation was involved. 
Instead, the program "causes the computer to respect the presumption of each factor as far as it 
can" (p. 137). The targeted factor extraction procedure was used to extract the two theory-based 
interest dimensions. The dimensions were defined by using the Cartesian coordinates in the 
hexagonal arrangement of Holland types to specify the relative sizes of the correlations expected 
between the six scales and the theory-based dimensions (Prediger, 1982). If the data/ideas and 
things/people dimensions fit the data perfectly, they should account for the maximum amount of 
variance that can be accounted for by any two interest dimensions. Principal component analyses 
provided the comparative data. 
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When all interest inventory items are scored in the same direction (e.g., dislike, indifferent, 
like), as in UNIACT, scores on interest scales are affected by response style. In the vocational 
interest domain, response style refers to systematic, non-interest-related differences in responses 
across counselees (e.g., some tend to choose like more often than others do) regardless of basic 
interest area (Prediger & Swaney, 1995). The primary identifying feature of a response style factor 
is that, in the initial factor matrix, aU interest scales have relatively high loadings on the factor in 
question. Based on numerous intercorrelation matrices for instmments assessing Holland's types, 
Prediger (1982) showed that the response style factor often accounts for 40% or more of the total 

variance. 

Principal component analyses were conducted on the four UNIACT scale intercorrelation 
matrices: Grade 9 CD-i, Grade 9 paper-and-pencil. Grade 1 1 CD-i, and Grade 1 1 paper-and-pencil. 
For comparison purposes, principal component analyses for the CD-i and paper-and-pencU 
presentations are shown side by side in Tables 9 and 10. Each of the principal component analyses 
yielded three orthogonal factors with eigenvalues of approximately one or greater than one. 
Together, the three factors accounted for 76% (Grade 9 paper-and-pencU), 74% (Grade 9 CD-i), 
73% (Grade 11 paper-and-pencU), and 72% (Grade 11 CD-i) of the total variance. Across the four 
principal component analyses, the first factor— the Response Set factor— accounted for 32% to 
39% of the variance. Across the four analyses, the pattern of factor loadings was very similar for 
corresponding factors. In each case, the pattern of loadings for the second and third factors was 
consistent with data/ideas and things/people dimensions, respectively. As noted earher, these two 
dimensions provide the basis for explaining degree of stmctural comparability across the two 
modes. After variance accounted for by response style was removed, the remaining total percentage 
of variance accounted for by the second and third factors ranged from 32% to 35% for Data/Ideas, 
and from 24% to 27% for Things/People, respectively. 
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The same four UNIACT scale intercorrelation matrices were subjected to the targeted factor 
extraction procedure. The plotted factor loadings are shown in Figures 3 and 4. The hexagonal 
configurations of UNIACT scales on the Data/Ideas and Things/People factors were nearly identical 
for the paper-and-pencil and CD-i presentation modes. When plotted separately for males and 
females, the results were also very similar. The configurations approximated those reported by 
Swaney (1995) for two UNIACT national norming samples. 

Tables 11 and 12 summarize the results of the targeted factor analysis. Across the four 
analyses, the factor loadings were very similar for corresponding factors. After variance associated 
with response style was removed, the remaining total percentage of variance accounted for by the 
theory-based dimensions was identical, or nearly identical, to that obtained via principal component 
analyses. 

As noted earlier, IWRA scales are not independent; some of the items appear in more than 
one scale. Therefore, principal component and targeted factor analyses were not conducted at the 
scale level. However, IWRA principal component analyses were mn at the item level, and the 
results suggest similar factor structure across modes within grade level. 

Quantitative Comparability 

UNIACT. Examining scale score differences for the two modes of presentation serves 
two purposes. Such an examination compares the extent to which the CD-i and paper-and-pencil 
scores are equivalent and allows for the identification of possible problematic CD-i items. 
UNIACT scale score mode of presentation differences were determined for the total group and 
for males and females. 

UNIACT scale score means and standard deviations for the two modes are shown in 
Table 13. For Grade 9, the average mean scores were 5.3 for both paper-and-pencil and CD-i. 
Examined separately by scale, differences in mean scores between paper-and-pencil and CD-i 
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ranged from -0.2 to 0.2. Across the six scales, the mean absolute difference was 0.1. For Grade 
11, three of the six UNIACT scale means were identical for the two modes. The mean 
differences for the three remaining scales ranged from -0.3 to 0.2. The mean absolute difference 
across the six scales was 0.1. These differences were extremely small relative to the 

corresponding scale standard deviations. 

Male and female UNIACT mean raw scores are shown in Tables 14 and 15. The mode of 
presentation mean differences were extremely small. For Grade 9 males and females, the mean 
differences across the six scales ranged from -0.6 to 0.8 and -0.8 to 0.3, respectively. The absolute 
mean CD-i and paper-and-pencU differences for males (0.5) and females (0.4) were small. The 
mean differences across the six scales for Grade 1 1 males and females ranged from -0.5 to 1.3 and 
-0.4 to 0.7, respectively. The absolute mean differences for males (0.5) and females (0.3) also were 

small. 

UNIACT items were carefully chosen to elicit sunilar response distributions from males and 
females (Prediger & Johnson, 1979; Swaney, 1995). Perspective on male-female score differences 
is provided by Dunnette’s (1966) table for TUton's (1937) measure of distribution overlap. Dunnette 
(1966) has suggested that two distributions are similar if their distributions overlap by more than 
75% to 80%. Male-female overlap values for Grade 9 were 94% (SC), 86% (AR), 72% (SS), 93% 
(BC), 96% (BO), and 76% (TE) (mean = 86%) for paper-and-pencil and 92% (SC), 89% (AR), 
72% (SS), 97% (BC), 96% (BO), and 77% (TE) (mean = 87%) for CD-i. Similar overlap values 
were obtained for Grade 11. The relative magnitudes of the six scale overlap percentages are highly 
consistent across the two modes of presentation. The ranges of percent overlap for the CD-i mode 
for Grade 9 (72%-97%) and Grade 11 (69%-98%) are also consistent with percent overlaps 
reported for UNIACT by Lamb and Prediger (1981) and Swaney (1995). 
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IWRA. For IWRA, there are two types of scaled scores to examine for mode of presentation 
mean differences. IWRA job cluster scale score mean differences are shown in Table 16. For 
Grades 9 and 1 1, the mean absolute across-mode differences for the six scales were 0.85 and 1.22, 
respectively. These differences were relatively small differences when compared to the magnitude 
of the scale standard deviations. 

Tables 17 and 18 present the IWRA mean scaled scores for the 15 abilities for the two 
modes. Again, for both samples, mean score differences were extremely small. The largest mean 
difference (0.5) was approximately one fourth of the corresponding standard deviations. 

The absence of appreciable differences for the two modes of presentation for UNIACT and 
IWRA suggests that the introduction of pictures and voice-over in the multimedia mode has little, if 
any, effect at the scale level. The paper-and-pencil and multimedia versions appear quantitatively 
comparable. 

Reliability 

UNIACT. Internal consistency reliability coefficients for the six UNIACT scales (see Table 
13) were comparable for the paper-and-pencil and CD-i modes. For Grade 9, coefficient alphas 
(Cronbach, 1951) ranged from .84 to .91 for CD- i and from .86 to .92 for paper-and-pencil. The 
corresponding Grade 11 alphas ranged from .82 to .91 for CD-i and from .84 to .91 for paper-and- 
pencil. 

IWRA. The coefficient alphas for the IWRA job cluster scales (see Table 16) ranged from 
.51 to .63 for CD-i and from .58 to .66 for paper-and-pencil for Grade 9. For Grade 1 1, coefficient 
alphas ranged from .47 to .60 for CD-i and from .49 to .67 for paper-and-pencil. The lower 
coefficients for the abUity scales may have occurred due to the sensitivity of coefficient alpha to the 
number of items in a scale (Cortina, 1993) and the fact that the items in the scales were not selected 
to be homogeneous. Each ability scale consists of only four abilities — albeit abilities shown to be 
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important to occupations in the corresponding job cluster (Prediger, 1992). Several studies (e.g., 
Prediger & Brandt, 1991; Prediger & Swaney, 1992) have demonstrated the validity of the ability 
scales for use in career exploration. 

S umma ry and Conclusions 

The American Psychological Association's Guidelines for Computer-Based Tests and 
Interpretations (1986) deal specifically with comparability of paper-and-pencil and computerized 
tests. By extension, these guidelines and the suggestions of Green (1991), Van de Vijver and 
Harsveld (1994), and King and Miles (1995) were used as a basis for determining the comparability 
of paper-and-pencil and multimedia versions of two assessments. At this point in time, the 
multimedia versions of UNIACT and IWRA appear to be unique; few of the aural/pictorial 
assessments in Table 1 combine written text, voice-overs, and pictures. In evaluating the 
qualitative comparability of verbal and pictorial forms of the 1 1 Vll , Tetreau and Trahan (1986) 
pfoyijjg ^ point of comparison. Tetreau and Trahan reported scale interconelations (i.e., .78 to .91) 
that were highly similar to those obtained in this smdy (see Table 6). 

The primary purpose of this smdy was to evaluate the qualitative and quantitative 
comparability of the multimedia and paper-and-pencil versions of UNIACT and IWRA. The high 
cross-mode scale correlations lend support to the qualitative comparability for both UNIACT and 
IWRA, and the absence of appreciable mean scale differences indicates that the paper-and-pencil 
and multimedia versions are quantitatively comparable. For UNIACT, there is additional evidence 
of the qualitative comparability. The pattern of scale intercorrelations generally corresponds to 
theoretical expectations. The two-dimensional plots of the factor scores are nearly identical for the 
two modes, and approximate the theory-based hexagonal configuration. 

Given the degree of qualitative and quantitative comparability of the multimedia and paper- 
and-pencil versions found here, it is appropriate to equate multimedia scores to existing paper-and- 
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pencil-based scores. The rescaling procedure is described in Appendix B. [It should be noted that 
ACT recently revised the scoring procedure for IWRA (norming at the scale level vs. the item 
level).] 

This finding of measured comparability in the scores based on the paper-and-pencil and 
multimedia versions of UNIACT and IWRA is important for users and practitioners. The 
multimedia versions of DISCOVER provide counselees with an enhanced career exploration 
experience. Special populations, such as individuals with disabilities or reading deficiencies, can 
use these assessments in career exploration and planning. Users and practitioners can have 
confidence in the rehability and validity of the multimedia versions. 

A potential limitation of this study is that the multimedia versions of the two assessments 
were administered in a group situation; students recorded their responses on an answer document. 
Use of an answer document was necessary to capture item responses. Also, this approach yielded 
desired sample sizes in an efficient manner. T 3 ^ically, counselees using DISCOVER Multimedia 
or DISCOVER for Windows would indicate their responses by pointing and chcking or by 
keystrokes, respectively. Future studies of comparability might compare DISCOVER (1995b) with 
DISCOVER for Windows (1997) versions of UNIACT and IWRA. 
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APPENDIX A 



CAREER GUIDANCE SURVEY 



Instructions for Completing the Answer Sheet 

Because your answer sheet will be processed by computer, please use a No. 2 pencil and print clearly. 
Blacken each circle completely, making sure all marks are dark. Completely erase any changes. Do not 
fold your answer sheet. 



Block A. Name : Print your name in the large boxes labeled Last Name, First Name, Ml (middle initial). 
Begin in the first box for each part of your name. Enter as much of your name as possible, using one box 
per letter. Do not extend any part of your name into the boxes reserved for another part of your name. 

Next, blacken the correct circles below each letter of your name. 

Block B. Date of Birth : Blacken the correct circle for your month. Then enter the numbers for your day 
and year in the boxes. 

Next, blacken the correct circles below each number. 

Block C : Skip this block. 

Block D. Sex : Blacken the correct circle--M for male, F for female. 

Block E. Racial/Ethnic Group : Which of the phrases below best describes your racial/ethnic group as 
generally recognized by your family and friends? Blacken the correct circle on the answer sheet. 



1 . African-Americ£tn/Black 

2. American Indian, Alaskan native 

3. Caucasian-AmericanAAihite 

4. Mexican-AmericeuVChicano 

5. Asian-American, Pacific Islander 

6. Puerto Rican, Cuban, other Hispanic origin 

7. Other 

8. 1 prefer not to respond. 



Block F. Grade : Blacken the correct circle for your grade level. 



Skip blocks G, H, and I. 



©1993 by American College Testing. Iowa City, Iowa All rights reserved. 
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Ability Self-Estimate inventory 

This inventory will help you estimate your standing on 15 important abilities. We will provide you with a report that 
relates your abilities to the four basic work tasks-working with DATA, PEOPLE, THINGS, and IDEAS. The report will 
also suggest occupations in line with your abilities. The suggestions will only be as good as the accuracy of your 
ability estimates. 

1st: For each ability, carefully read the examples of related activities. Then estimate your ability compared to 
persons vour own age . Use the numbers In the box below. 



5 = high (top 10%) 

4 = above average (upper 25%) 
3 = average (middle 50%) 

2 = below average (lower 25%) 
1 = low (bottom 1 0®/o) 



2nd: Blacken the circle for your estimate on your answer sheet. Erase completely any estimate you change. 



Ability 

1. Meeting People 



2. Helping Others 



3. Sales 



4. Leadership (Management) 



5. Organization 



Examples of Activities 

Talking with people; getting along with others; making a good 
impression. Consider your ability to help people feel at ease; to be courteous, 
pleasant, or informative; to remember names and faces. 

Caring for or teaching others; helping people with problems or decisions. 
Consider your ability to explain how to do something; to understand the ideas 
and feeling of others; to help someone feel better; to be tactful and patient. 

Influencing people to buy a product, service, or take a suggested course 
of action. Consider your ability to change someone's mind; to bargain; to 
make a sale; to persuade a group; AND how you did in speech, debate, 
distributive education, marketing, etc. 

Leading/managing people so that they work toward a common goal. 
Consider your ability to present ideas to a group; to motivate others and provide 
direction; to plan an event; to stick to a budget. 

Keeping track of tasks and details; doing things in a systematic way. 
Consider your ability to keep to a schedule; to see what needs to be done first, 
second, etc.; to store things (pictures, clippings, tools, etc.) so they are easy to 
find. 



6. Clerical Quickly and accurately doing tasks such as looking up information in 

catalogs or tables, sorting things, recording addresses or expenses, etc. 
Consider your ability to handle papenn^ork; to type; to complete forms accurately 
and neatly (e.g., an application); to catch errors. 

7. Mechanical Understanding everyday mechanical laws (e.g., warm air rises) and how 

simple mechanical things work (e.g., a lever, a pulley). Consider how easily 
you figure out how things work (toys, tools, appliances, etc.) and how to fix 
them; AND how you did in general science, industrial arts, home economics, 
shop. 
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5 = high (top 10%) 

4 = above average (upper 25%) 
3 = average (middle 50%) 

2 = below average (lower 25%) 
1 = low (bottom 10%) 



Ability 



8. Manual Dexterity 



9. Numerical 



10. Scientific 



11. Creative/Artistic 



12. Creative/Literary 



13. Reading 



14. Language Usage 



15. Spatial Perception 



Examples of Activities 

Making or repairing things easily and quickly with one's hands. Consider 
your ability to handle tools, appliances, or to assemble things (toys, furniture, 
picture frames, etc.); to do handicrafts; to handle or connect small objects; to 
use your coordination; AND how you did in industrial arts, home economics, 
shop. 

Doing arithmetic accurately and quickly; applying arithmetic (e.g., in 
formulas and word problems). Consider how you did in arithmetic when 
studied in various classes; your skill with a calculator, in keeping track of 
expenses, in figuring interest rates, or in finding the "best buy," etc. 

Understanding science laws; doing science course work. Consider your 
ability to use math rules and formulas; to understand articles or TV programs on 
science, health, or technology; AND how you did in classes such as general 
science, chemistry, biology, etc. 

Drawing, painting, playing a musical instrument, acting, dancing, etc. 
Consider how well you can express ideas, feelings, or moods through one or 
more of the performing arts; AND how you did in classes such as art, music, 
dance-either in or out of school. 

Expressing ideas or feelings through writing. Consider your ability to write 
interesting letters to friends or family members; to write reports, explanations of 
events, opinions, etc.; AND how you did on themes in English and in other 
classes. 

Reading and understanding factual material (for example, in a textbook or 
manual). Consider your ability to finish reading assignments (speed and 
understanding); to read directions or warranties for a tool, appliance, TV, etc.; 
to follow news stories in magazines, editorials, etc. 

Recognizing correct and incorrect uses of the English language 
(grammar, punctuation, etc.). Consider your ability to write and speak 
correctly; to organize and present your thoughts; AND how you did in classes 
such as English and speech. 

Looking at a drawing of an object (e.g., a house, coat, tool) and picturing 
in your mind how it would look from different sides. Consider your ability to 
"read" and explain blueprints, clothing patterns, etc.; to see how things could fit 
within the available space (a box, room, closet, etc.). 



NEXT: Look over your estimates to see if they show how your abilities rank-top to bottom. Carefully erase any 
estimate you wish to change and blacken the circle for your revised estimate. 
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The ACT Interest Inventory 

The things you like to do now can give you clues about jobs you might like. This inventory will help you identify occupations 
you may want to explore. We will provide you with a report that relates your interests to the four basic work tasks-working 
with DATA, PEOPLE, THINGS, and IDEAS. The report will also suggest occupations in line with your Interests. 

Show how much you would like doing each of the activities listed below. Mark an answer to an activity even if you are 
uncertain how you feel about it. Consider whether you would like or dislike the activity, not your ability to do it. 

For each activity, choose one of the answers below. Blacken the circle on your answer sheet that contains the letter for your 
answer. Try to answer like or dislike as often as possible. 



I would dislike doing this activity D 

I am indifferent I 

I would like doing this activity L 



1 . Learn about star formations 

2. Sketch and draw pictures 

3. Help someone make an Important decision 

4. Conduct a meeting 

5. Count and sort money 

6. Build a picture frame 

7. Learn how the brain works 

8. Compose or arrange music 

9. Give first aid to an injured person 

1 0. Develop new rules or policies 

11. T ake inventory in a store 

12. Rxatoy 

13. Explore a science museum 

14. Make creative photographs 

15. Show children how to play a game or sport 

16. Work in a poiiticai campaign 

17. Write payroll checks 

18. Run a lawn mower 

19. Attend the lecture of a well-known scientist 

20. Write short stories 

21 . Work on a community improvement project 

22. Present information before a group 

23. Set up a bookkeeping system 

24. Watch for forest fires 

25. Study biology 

26. Read about the writing style of modem authors 

27. Help a newcomer meet people 

28. Discuss a misleading advertisement with a salesperson 

29. Prepare a budget for a club or group 

30. Build furniture 

31 . Measure chemicals in a test tube 

32. Prepare drawings to illustrate a magazine story 

33. Take part in a small group discussion 

34. Plan work for other people 

35. Balance a checkbook 

36. Learn to cut and polish gemstones 

37. Read about a new surgical procedure 

38. Write a movie script 

39. Find out how others believe a problem can be solved 

40. Explain legal rights to people 

41 . Sort, count, and store supplies 

42. Repair damage to a tree after a storm 

43. Study plant diseases 

44. Select music to play for a local radio station 

45. Help rescue someone in danger 



46. Demonstrate a new product 

47. Plan a monthly budget 

48. Design a bird feeder 

49. Read books or magazines about new scientific findings 

50. Play jazz in a combo 

51 . Help settle an argument between friends 

52. Campaign for a political office 

53. Rnd errors in a financial account 

54. Engrave lettering or designs on a trophy or plaque 

55. Study chemistry 

56. Draw cartoons 

57. Give directions to visitors 

58. Publicize a show or athletic event 

59. Rgure shipping costs for catalog orders 

60. Operate a slide or movie projector 

61 . Use a microscope or other lab equipment 

62. Design a metal sculpture 

63. Help friends with their problems 

64. Conduct business by phone 

65. Make charts or graphs 

66. Learn how to repair a typewriter 

67. Read about the origin of the earth, sun. and stars 

68. Play in a band 

69. Teach people a new hobby 

70. Interview workers about company complaints 

71 . Calculate the interest on a loan 

72. Watch a technician repair a television 

73. Observe and classify butterflies 

74. Write reviews of Broadway plays 

75. Help people during emergencies 

76. Hire a person for a job 

77. Keep expense account records 

78. Prune plants and shrubs 

79. Study the effects of vitamins on animals 

80. Design a poster for an event 

81 . Entertain others by telling jokes or stories 

82. Manage a small business 

83. Look for errors in the draft of a report 

84. Shelve books in a library 

85. Learn how birds migrate 

86. Play a musical instrument 

87. Give a tour of an exhibit 

88. Conduct a door-to-door opinion poll 

89. Operate office machines 

90. Inspect products for defects 




BEST COPY AVAIUBLE 



36 



29 



APPENDIX B 
Equating 

For both UNIACT and IWRA, the scores of the multimedia version were equated to the paper- 
and-pencil version using an equipercentile method of equating. In the 9th-grade sample 606 students 
completed both UNIACT and IWRA via the two modes of administration in a counterbalanced design. 
In the 1 Ith-grade sample, 416 students completed IWRA and UNIACT via the two modes of 
administration. 

In equating the multimedia to the paper-and-pencil forms, cumulative percent distributions of 
scores for both forms were prepared and the data were used to locate points below which equal 
proportions of the distributions fell. A conversion curve was then drawn and the equivalent scores 
were read from this curve. Since this procedure does not usually result in the smooth curve that would 
be expected to display the relationship between the scores on two forms of a test, a method of 
analytically smoothing using cubic spline functions (Kolen, 1984) was employed. The conversion 
table that results from this process was then used to transform scores from the multimedia mode of 
administration to scale scores. 
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TABLE 2 



Schools, Testing Dates, and Sample Sizes 



School 


Type of 
community 


Testing 

interval 

(days) 


Students 
in study 


Group X 

Number of 
classrooms 


Sample Sizes 

Group Y 

Number of 
N classrooms 


N 








Grade 9 


a 








1 


Rural 


11 


56 


1 


31 


2 


25 


2 


Rural 


15 


62 


1 


24 


2 


38 


3 


Urban 


6 


43 


1 


22 


1 


21 


4 


Urban 


7 


37 


1 


19 


1 


18 


5 


Urban 


8 


55 


1 


15 


2 


40 


6 


Urban 


8 


56 


2 


37 


1 


19 


7 


Rural 


7 


65 


2 


36 


2 


29 


8 


Urban 


7 


32 


1 


15 


1 


17 


9 


Urban 


7 


200 


5 


101 


5 


99 








Grade ll'’ 








1 


Suburban 


7 


98 


3 


52 


3 


46 


2 


Urban 


14 


26 


1 


16 


1 


10 


3 


Suburban 


14 


106 


3 


53 


3 


53 


4 


Urban 


9 


186 


4 


85 


5 


101 



®Group X iV = 300; Group Y N = 306. '’Group X = 206; Group Y N = 210. 




TABLE 3 

Selected Characteristics of Grade 9 and Grade 11 Group X and Y Samples 
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non-white. Twenty-nine of the 67 were in Group X, and 38 of the 67 were in Group Y. Of the 389 
Grade 1 1 students responding to the racial/ethnic question, 14 (4%) reported they were non-white. The 
non- white students were equally divided between Group X and Group Y. 



Time and Group Differences in UNIACT Stanine Scores 
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“Difference between Time 1 and Time 2 mean UNIACT scores. ’’Group XN = 300; Group Y N = 306. '’Group XN = 
206; Group Y A =210. 



TABLE 5 

Time and Group Differences in IWRA Job Cluster Scores 
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TABLE 6 

Cross-Mode Scale Correlations for UNIACT and IWRA 



Scale 


UNIACT 




IWRA 


Grade 9“ 


Grade 1 1*" 


Grade 9“^ 


Grade ll“ 


SC 


.87 


.87 


.78 


.75 


AR 


.84 


.90 


.72 


.78 


SS 


.79 


.81 


.74 


.69 


BC 


.80 


.78 


.72 


.77 


BO 


.80 


.82 


.72 


.74 


TE 


.81 


.82 


.71 


.76 



Note. The testing interval ranged from 6-15 days for Grade 9 and 7-14 days for 
Grade 11. 

"A^=606. ‘’iV = 416. 601-605. “iV = 413-415. 
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TABLE 7 

Correlations Among UNIACT Scale Scores: CD-i and Paper-and-Pendl Modes 



Scale 






UNIACT Scale 






SC 


AR 


SS 


BC 


BO 


TE 






Grade 9“ 








SC 


~ 


.25 


.24 


.16 


.01 


.35 


AR 


.26 


— 


.37 


.23 


-.05 


.22 


SS 


.19 


.38 


— 


.54 


.23 


.16 


BC 


.17 


.26 


.56 


— 


.51 


.18 


BO 


.04 


-.01 


.24 


.46 


— 


.30 


TE 


.30 


.30 


.20 


.21 


.29 


— 






Grade 11*’ 








SC 


— 


.18 


.23 


.13 


.04 


.28 


AR 


.18 


— 


.39 


.09 


-.15 


.32 


SS 


.23 


.32 


— 


.41 


.14 


.18 


BC 


.14 


.05 


.42 


— 


.49 


.07 


BO 


-.02 


-.20 


.15 


.39 


— 


.24 


TE 


.29 


.34 


.17 


.04 


.21 


— 



Note. Paper-and-pencil correlations are shown above the diagonals; CD-i correlations are below 
the diagonals. 

“N = 606. *’N = 416. 
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TABLE 8 

Correlations Among IWRA Job Cluster Scale Scores: 
CD-i and Paper-and-Pencil Modes 



Job Cluster Ability Scale 



Scale 


SC 


AR 


SS 


BC 


BO 


TE 








Grade 9“ 








SC 


— 


.56 


.51 


.65 


.62 


.60 


AR 


.53 


— 


.65 


.54 


.47 


.39 


ss 


.53 


.64 


— 


.59 


.54 


.16 


BC 


.68 


.52 


.60 


— 


.70 


.38 


BO 


.57 


.43 


.56 


.72 


— 


.46 


TE 


.57 


.37 


.14 


.38 


.38 


— 








Grade 1 1*’ 








SC 


— 


.49 


.45 


.63 


.66 


.53 


AR 


.47 


— 


.61 


.43 


.37 


.35 


SS 


.46 


.60 


- 


.54 


.43 


.10 


BC 


.64 


.44 


.52 


— 


.67 


.35 


BO 


.61 


.35 


.47 


.70 


— 


.51 


TE 


.55 


.33 


.04 


.35 


.42 






Note. Paper-and-pencil are shown above the diagonals; CD-i correlations are below the 
diagonals. 

“A =599-606. V = 413-416. 



53 

o 

ERIC 



Grade 9 UNIACT Scale Loadings on the First Three Principal Components 
for Two Principal Component Analyses (Paper-and-Pencil and CD-i) 
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Grade 11 UNIACT Scale Loadings on the First Three Principal Components 
for Two Principal Component Analyses (Paper-and-Pencil and CD-i) 
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TABLE 11 

Grade 9 UNIACT Scale Targeted Factor® Loadings on the First Three Factors 
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TABLE 12 

Grade 11 UNIACT Scale Targeted Factor® Loadings on the First Three Factors 
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TABLE 13 

Mean UNIACT Stanine Scores and Reliabilities 



Scale 


CD-i 


Paper-and-pencil 


Difference*’ 


Reliability* 


Mean 


SD 


Mean 


SD 


CD-i 


ppc 








Grade 9 










SC 


5.4 


2.0 


5.3 


2.2 


-0.1 


.91 


.92 


AR 


5.8 


2.1 


5.6 


2.1 


-0.2 


.87 


.87 


ss 


5.6 


2.3 


5.6 


2.3 


0.0 


.87 


.87 


BC 


5.2 


2.0 


5.3 


2.1 


0.1 


.84 


.86 


BO 


4.8 


2.2 


5.0 


2.4 


0.2 


.90 


.91 


TE 


5.0 


2.2 


5.2 


2.2 


0.2 


.85 


.86 








Grade 1 1 








SC 


5.4 


2.0 


5.3 


2.1 


-0.1 


.91 


.91 


AR 


5.7 


2.1 


5.4 


2.2 


-0.3 


.87 


.88 


SS 


5.4 


2.2 


5.6 


2.2 


0.2 


.83 


.85 


BC 


5.2 


2.0 


5.2 


2.1 


0.0 


.82 


.84 


BO 


4.6 


2.2 


4.6 


2.3 


0.0 


.90 


.91 


TE 


4.6 


2.3 


4.6 


2.3 


0.0 


.87 


.86 


Note. For means and standard deviations, N = 606 and 416 for Grades 9 and 11, 



respectively. For CD-i and paper-and-pencil reliabilities, Ns = 604 and 596, respectively, for 
Grade 9; and iVs = 413 and 410, respectively, for Grade 11. 

“Cronbach's (1951) coefficient alpha. '’Mean differences. ‘^Paper-and-pencil. 
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TABLE 14 

Grade 9 Male and Female Mode of Presentation Differences for UNIACT 



Scale 


CD-i 


Paper-and-Pencil 


Diff 


Mean 


SD 


Mean 


SD 






Males {N = 


301) 






SC 


14.9 


8.6 


14.2 


9.1 


0.7 


AR 


16.7 


7.1 


15.9 


7.3 


0.8 


ss 


18.8 


7.3 


19.0 


7.2 


-0.2 


BC 


14.4 


6.9 


14.4 


7.1 


0.0 


BO 


11.2 


7.7 


11.8 


8.1 


-0.6 


TE 


14.6 


7.2 


15.1 


7.5 


-0.5 






Females {N - 


= 305) 






SC 


13.1 


8.9 


12.9 


8.8 


0.2 


AR 


18.9 


7.8 


18.6 


7.8 


0.3 


SS 


23.4 


5.4 


23.5 


5.5 


-0.1 


BC 


15.0 


7.0 


15.6 


7.4 


-0.6 


BO 


12.0 


8.4 


12.8 


8.9 


-0.8 


TE 


10.4 


6.9 


10.8 


7.0 


-0.4 



Note. Means and SDs based on raw scores ranging from 0-30. 

“Mean differences. Male and female mean absolute differences were 0.5 and 0.4, respectively. 
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TABLE 15 

Grade 11 Male and Female Mode of Presentation Differences for UNIACT 



Scale 


CD-i 


Paper-and-pencil 


Diff 


Mean 


SD 


Mean 


SD 






Males {N = 


233) 






SC 


14.8 


8.8 


14.0 


8.9 


0.8 


AR 


17.5 


8.0 


16.2 


8.0 


1.3 


SS 


19.8 


6.8 


20.3 


7.0 


-0.5 


BC 


15.6 


6.9 


15.6 


7.4 


0.0 


BO 


12.2 


8.3 


n.i 


8.5 


-0.5 


TE 


13.5 


7.6 


13.6 


7.2 


-0.1 






Females {N ■ 


= 183) 






SC 


13.7 


9.2 


13.3 


9.1 


0.4 


AR 


18.7 


7.9 


18.0 


8.3 


0.7 


SS 


24.5 


5.0 


24.5 


5.3 


0.0 


BC 


16.0 


7.0 


15.8 


7.2 


0.2 


BO 


11.0 


8.8 


10.7 


8.8 


0.3 


TE 


8.9 


7.4 


9.3 


7.5 


-0.4 



Note. Means and SDs based on raw scores ranging from 0-30. 



*Mean difference. Male and female mean absolute differences were 0.5 and 0.3, respectively. 
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TABLE 16 

IWRA Job Cluster Scale Scores and Reliabilities 



Scale 


CD-i 


Paper-and-pencil 


Difference'’ 


Reliability“ 


Mean 


SD 


Mean 


SD 


CD-i 


ppc 








Grade 9 










SC 


51.5 


15.4 


51.2 


15.4 


-.03 


.60 


.59 


AR 


50.7 


13.7 


51.7 


14.3 


1.0 


.51 


.59 


SS 


44.1 


14.8 


45.4 


16.0 


1.3 


.60 


.66 


BC 


48.6 


14.4 


48.5 


15.3 


-0.1 


.56 


.58 


BO 


46.4 


15.2 


45.1 


15.1 


-1.3 


.63 


.62 


TE 


45.2 


13.5 


46.3 


14.1 


1.1 


.56 


.60 








Grade 1 1 










SC 


50.4 


15.1 


50.7 


14.3 


-0.3 


.60 


.53 


AR 


49.9 


14.3 


51.8 


14.5 


1.9 


.57 


.63 


SS 


44.3 


13.9 


46.6 


14.7 


2.3 


.58 


.62 


BC 


48.9 


13.5 


49.9 


13.8 


1.0 


.47 


.49 


BO 


45.6 


13.9 


46.2 


13.9 


0.6 


.57 


.56 


TE 


44.0 


13.5 


45.2 


14.2 


1.2 


.60 


.67 



Note. For means and standard deviations, N = 602-606 for Grade 9, and N = 414-416 for 
Grade 11. For CD-i and paper-and-pencil reliabilities, Ns = 604 and 596, respectively, for 
Grade 9; and A^s = 413 and 410, respectively, for Grade 11. 



“Cronbach’s (1951) coefficient alpha. '’Mean differences. ‘’Paper-and-pencil. 
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TABLE 17 

Grade 9 IWRA Mean Scaled Scores 



Ability 


CD-i 




Paper-and-pencil 


Difference“ 


Mean 


SD 


Mean 


SD 


Meeting People 


4.1 


2.1 


4.5 


2.3 


0.4 


Helping Others 


4.2 


2.1 


4.4 


2.2 


0.2 


Sales 


4.5 


2.1 


4.6 


2.3 


0.1 


Leadership 


5.0 


2.2 


4.9 


2.4 


-0.1 


Organization 


4.6 


2.3 


4.6 


2.3 


0.0 


Clerical 


4.1 


2.0 


3.6 


2.0 


-0.5 


Mechanical 


4.0 


1.9 


4.3 


1.9 


0.3 


Manual Dexterity 


4.4 


1.8 


4.4 


1.9 


0.0 


Numerical 


5.1 


2.3 


5.0 


2.3 


-0.1 


Scientific 


5.8 


2.3 


5.8 


2.4 


0.0 


Artistic 


5.6 


2.0 


5.6 


2.0 


0.0 


Literary 


4.8 


2.3 


4.9 


2.2 


0.1 


Reading 


4.6 


2.3 


4.4 


2.3 


-0.2 


Language Usage 


4.7 


2.2 


4.8 


2.2 


0.1 


Spatial Perception 


5.1 


2.2 


5.2 


2.1 


0.1 



Note. iV=606. The overall paper-and-pencil and CD-i means are 4.73 
and 4.71, respectively. 

“Mean difference. Mean absolute across the 15 means is 0.15. 
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TABLE 18 

Grade 11 IWRA Mean Scaled Scores 



Ability 


CD-i 




Paper-and-pencil 


Difference® 


Mean 


SD 


Mean 


SD 


Meeting People 


4.3 


2.0 


4.6 


2.2 


0.3 


Helping Others 


4.2 


2.0 


4.7 


2.1 


0.5 


Sales 


4.7 


2.1 


4.8 


2.3 


0.1 


Leadership 


5.3 


2.1 


5.1 


2.2 


-0.2 


Organization 


4.7 


2.2 


4.7 


2.3 


0.0 


Clerical 


4.1 


1.8 


3.7 


1.9 


-0.4 


Mechanical 


3.7 


1.9 


4.0 


1.9 


0.3 


Manual Dexterity 


4.2 


1.9 


4.1 


1.9 


-0.1 


Numerical 


4.8 


2.3 


4.9 


2.3 


0.1 


Scientific 


5.8 


2.4 


6.0 


2.4 


0.2 


Artistic 


5.2 


2.2 


5.3 


2.2 


0.1 


Literary 


5.1 


2.3 


5.2 


2.2 


0.1 


Reading 


4.5 


2.2 


4.2 


2.2 


-0.3 


Language Usage 


4.6 


2.1 


5.0 


2.1 


0.4 


Spatial Perception 


5.0 


2.1 


5.1 


2.0 


0.1 



Note. A = 416. The overall paper-and-pencil and CD-i means are 4.77 
and 4.68, respectively. 



“Mean difference. Mean absolute across the 15 means is 0.21. 
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Grade 9 





Time One 


Time Two 


Group X 


I(CD-i) + A(PP) 


I(PP) + A(CD-i) 


Group Y 


I(PP) + A(CD-i) 


I(CD-i) + A(PP) 



Grade 9 Modified 





Time One 


Time Two 


Group X 


I(CD-i) + A(PP) 


A(CD-i) + I(PP) 


Group Y 


A(CD-i) + I(PP) 


I(CD-i) + A(PP) 



Grade 11 





Time One 


Time Two 


Group X 


I(CD-i) + A(PP) 


A(CD-i) + I(PP) 


Group Y 


A(CD-i) + I(PP) 


I(CD-i) + A(PP) 



Note. I = interests; A= ability self-estimates; PP = paper-and-pencil; 
CD-i = Compact Disc-Interactive. 



Figure 1. Research designs. 
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Figure 2. Relationship between UNIACT scales and Data/Ideas and Things/People Work Task 
Dimensions. Holland types corresponding to UNIACT scales are shown in parentheses. 
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Data 




Figure 3. Plot of UNIACT theory-based factor loadings for Grade 9. 
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Data 




Figure 4. Plot of UNIACT theory-based factor loadings for Grade 1 1 . 
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